ASSESSING ATTRITION BIAS —- ADDENDUM 


A. INTRODUCTION 


This document serves as an addendum to the existing attrition white paper (“Assessing 


Attrition Bias” version 2.1) previously used for the WWC Procedures and Standards Handbook, 
version 2.1. This document does not recommend any substantive changes to the version 2.1 
attrition bounds. Instead, this addendum presents additional empirical information and sensitivity 


analyses to support the existing attrition bounds. 


B. IDENTIFYING MODEL PARAMETERS USING AN ADDITIONAL STUDY 
The “Assessing Attrition Bias” white paper used data from three large-scale randomized 
trials conducted by Mathematica Policy Research for IES. These studies examined outcomes, 
intervention, and populations relevant to the WWC and were thus seen as good sources from 
which to identify empirical model parameters to inform the attrition standard: 
e Evaluation of the 21" Century Community Learning Centers 
e Evaluation of Education Technologies in Reading and Mathematics! 


e Evaluation of Supplemental Reading Comprehension Interventions 


For this supplement, we also examined data from a fourth study to incorporate additional 
data into the empirical investigation to improve the generalizability of the attrition standard: 


e Evaluation of Teachers Trained Through Different Routes to Certification” 


' This study had distinct interventions that were implemented in grades 1, 4, 6, and 9, with random assignment 
occurring separately by grade level. Therefore, we calculated parameter values separately by grade level in this 
study. 


Table A.1 presents the empirical values of key quantities described in the attrition white 
paper. For both the treatment and control conditions, this table presents information on study 
response rates (P; and P,), respondent-nonrespondent differences in baseline scores (A; and A,), 
and the correlation coefficients between an individual’s propensity to respond and his or her 
baseline test score (oq; and a,). For the original three studies (excluding the new Teacher 
Certification Routes study), the values are identical with those shown in Table 2 in the attrition 
white paper. The following text summarizes the empirical values observed after including the 
Teacher Certification Routes results, noting where summary statistics differed from the 


information in the original analysis. 


TABLE A.1. RESPONSE RATES, RESPONDENT—NONRESPONDENT DIFFERENCES IN 
BASELINE TEST SCORES, AND IMPLIED CORRELATIONS BETWEEN TEST SCORES 


AND THE PROPENSITY TO RESPOND FROM FOUR RANDOMIZED TRIALS 


21st Century 0.81 0.02 0.01 0.83 0.10 0.06 
Education Technology 
1st Grade 0.91 0.46 0.23 0.90 0.35 0.18 
4th Grade 0.87 0.40 0.21 0.90 0.51 0.26 
6th Grade 0.88 0.54 0.28 0.90 0.44 0.23 
9th Grade 0.80 0.18 0.10 0.76 0.28 0.16 
Reading Comprehension 0.89 0.31 0.16 0.88 0.32 0.17 
Teacher Certification Routes 
Vocabulary 0.90 0.20 0.10 0.91 -0.14 -0.07 
Math 0.90 0.16 0.08 0.91 -0.04 -0.02 
(continued) 


* This study allowed for an estimation of model parameters for two outcomes (Vocabulary and Mathematics). 
Therefore, we calculated parameter values separately for each outcome. 


Across these studies, the higher value in each pair of a’s ranges from 0.06 to 0.28, with a 
mean of 0.17 (the mean was 0.19 before adding the new estimates). In addition, the difference 
between the higher and lower value of o ranges from 0.01 to 0.17, with a mean of 0.07 (the mean 
was 0).05 before adding the new estimates). 

Of the various sets of assumptions for o; and a, presented in Table 1 of the attrition white 
paper, the assumptions in the first column of Table 1 (q = 0.27 and a, = 0.22) still appear 


consistent with the empirical estimates of a. 


C. SENSITIVITY OF THE RESULTS TO ALTERNATE VALUES OF KEY 
PARAMETERS 

In this section, we examine the sensitivity of expected bias at the attrition boundaries shown 
in Figure | of the attrition white paper to variation in values for o, and a.. The purpose of this 
analysis is to assess the implications of errors in selecting values for these parameters. If small 
differences in these parameter values lead to substantial differences in expected bias at the 
boundary, then potentially costly efforts to develop better parameter estimates may be warranted. 
However, if these differences are not substantial, then the costs of adopting new boundaries may 
exceed the benefit. 

In this analysis, we hold the attrition boundaries fixed at the locations shown in Figure A.1 
below (that is, the existing attrition standard) and then calculate the average bias across all points 
along a boundary, given alternative values for o and a. Table A.2 shows the expected bias at 
the liberal attrition boundary (the boundary between the red and yellow regions in Figure A.1) 
and Table A.3 shows the expected bias at the conservative boundary (the boundary between the 


yellow and green regions in Figure A.1). 


FIGURE A.1. TRADEOFFS BETWEEN OVERALL AND DIFFERENTIAL ATTRITION 


Differential Attrition 


Overall Attrition 


We note two findings from Tables A.2 and A.3. First, the difference between o; and GQ, is 
much more important than the /eve/ of o and o,. This can be seen in both tables as bias increases 
faster moving vertically down the table than it does moving horizontally across the table. 
Second, within the range of the previously described empirical estimates of these correlations 
(indicated by the shaded cells in both tables), bias never exceeds 0.08 standard deviations at the 
conservative boundary and never exceeds 0.11 standard deviations at the liberal boundary. 


Outside of the empirical range, bias can be noticeably higher. 


TABLE A.2. SENSITIVITY OF EXPECTED BIAS AT THE LIBERAL ATTRITION 


BOUNDARY WITH RESPECT TO ATTRITION MODEL PARAMETER VALUES 


Overall Correlation Between Outcomes and Attrition 


Difference in 
Correlation Between 

Treatment and 

Control Groups 0.1 0.2 0.3 0.4 0.5 
0.05 0.04 0.05 0.05 0.06 0.07 
0.10 0.06 0.07 0.08 0.09 0.10 
0.15 0.09 0.10 0.11 0.12 0.13 
0.20 0.12 0.13 0.13 0.14 0.15 
0.25 0.14 0.15 0.16 0.17 0.18 


Note: Shaded cells correspond to the range of correlations estimated using data from past studies. 


TABLE A.3. SENSITIVITY OF EXPECTED BIAS AT THE CONSERVATIVE ATTRITION 


BOUNDARY WITH RESPECT TO ATTRITION MODEL PARAMETER VALUES 


Overall Correlation Between Outcomes and Attrition 


Difference in 
Correlation Between 

Treatment and 

Control Groups 0.1 0.2 0.3 0.4 0.5 
0.05 0.03 0.03 0.04 0.04 0.05 
0.10 0.05 0.06 0.06 0.07 0.07 
0.15 0.07 0.08 0.08 0.09 0.10 
0.20 0.10 0.10 0.11 0.11 0.12 
0.25 0.12 0.12 0.13 0.14 0.14 


Note: Shaded cells correspond to the range of correlations estimated using data from past studies. 


Because the attrition standard is expected to guard against bias in excess of 0.05 standard 
deviations, it is important that the current attrition standard maintain this tolerance in most 
studies. Given that in six of the eight empirical estimates, the difference in the correlation 
between the treatment and control groups was 0.06 or less (shown in Table A.1), it appears that 
the first row of Tables A.2 and A.3 may be the most appropriate lens on which to focus the 
appropriateness of the current standard. And given that in all of the shaded cells in those rows, 
the expected bias is less than or equal to 0.05 standard deviations, the current attrition standards 
appear to be appropriate and robust. As a result, the WWC recommends that the existing attrition 


standards continue to be used under version 3.0 of the Procedures and Standards Handbook. 


